1. Technical Field
One or more embodiments relate generally to systems and methods for operation automation. More specifically, one or more embodiments relate to systems and methods of automating user operations based on graphical user interface screenshots.
2. Background and Relevant Art
Automation software allows users to perform one or more operations without manually performing each of the operations. Performing one or more operations without requiring manual input from a user, allows users to quickly and easily accomplish tasks. In graphical user interfaces, operations can include mouse operations and/or keyboard operations for interacting with graphical user interface elements (e.g., folders and files) and/or applications within the graphical user interface. Automating operations in association with a graphical user interface can improve a user's productivity and efficiency. Simplifying such tasks can reduce manual repetitions of simple and/or complex operations, thereby reducing the time to perform the tasks.
In order to automate operations in a computing environment, conventional automation software typically interacts with the computing device using a specialized programming language to instruct the device to perform the operations. Some conventional automation software requires users to have extensive knowledge of the specialized programming language to be able to correctly automate the operations. For example, automation software can require a user to know syntax, inputs, outputs, and/or other information associated with the programming language for each operation. People without knowledge of the specific programming language may not be able to perform automation for even simple operations.
Furthermore, even users with knowledge of a specialized programming language may have difficulty setting up a script or macro for performing automated operations. Specifically, scripts and macros that control graphical user interface elements refer to the graphical user interface elements by name. Unfortunately, such names may be unknown, unfamiliar, or even unavailable to a user. Alternatively, some conventional automation methods allow a user to control graphical user interface elements using location of the elements. Such methods can be problematic and subject to errors as graphical user interface elements often change location.
In some instances, a user may want to verify that an automated process is performing each operation correctly. Conventional automation methods often require a user to debug the automated process by manually verifying each step individually as the user sets up the automation. Consequently, debugging automated processes can be time consuming, especially for automated processes that include a large number of automated operations or that include complex operations.
These and other disadvantages may exist with respect to conventional operation automation techniques.